Army  Research  Laboratory  ____  ___ 


Assessment  of  Crew  Workload  for  the 
RAH-66  Comanche  Force 
Development  Experiment  1 


David  B.  Durbin 


. 


ARL-TN-183 


OCTOBER  2001 


20011105  OH 


Approved  for  public  release;  distribution  is  unlimited 


Nintendo®  is  a  registered  trademark  of  Nintendo  of  America. 
Polhemus™  is  a  trademark  of  Rockwell  Collins. 

ProView  50™  is  a  trademark  of  Kaiser  Electro-Optics,  Inc. 


The  findings  in  this  report  are  not  to  be  construed  as  an  official  Department  of  the  Army  position 
unless  so  designated  by  other  authorized  documents. 

Citation  of  manufacturer’s  or  trade  names  does  not  constitute  an  official  endorsement  or  approval  of 

the  use  thereof. 

Destroy  this  report  when  it  is  no  longer  needed.  Do  not  return  it  to  the  originator. 


Army  Research  Laboratory 

Aberdeen  Proving  Ground,  MD  21005-5425 


ARL-TN-183 


October  2001 


Assessment  of  Crew  Workload  for  the 
RAH-66  Comanche  Force  Development 
Experiment  1 


David  B.  Durbin 

Human  Research  &  Engineering  Directorate 


Approved  for  public  release;  distribution  is  unlimited. 


i 


Abstract 


Crew  workload  was  assessed  during  the  RAH-66  Comanche  Force 
Development  Experiment  (FDE)  1.  The  purpose  was  to  determine  if 
(a)  the  pilots  experienced  tolerable  and  comparable  workload  levels 
when  flying  the  aircraft  versus  operating  the  mission  equipment 
package  (MEP),  and  (b)  workload  levels  contribute  to  a  need  to 
"battle  roster"  Comanche  pilots.  Workload  data  were  collected  via 
the  Bedford  Workload  Rating  Scale  and  a  cockpit  controls  and 
displays  usability  questionnaire. 

Results  of  the  assessment  indicate  that  (a)  workload  was  tolerable  for 
the  pilots,  (b)  workload  was  moderately  higher  for  the  pilots  when 
they  operated  the  MEP,  and  (c)  there  is  no  compelling  need  to 
prescribe  battle  rostering  because  pilots  experienced  only  moderate 
differences  in  workload  when  they  flew  the  aircraft  versus  when  they 
operated  the  MEP. 
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Executive  Summary 


An  assessment  of  crew  workload  was  conducted  during  the  RAH-66  Comanche 
Force  Development  Experiment  (FDE)  1.  Workload  data  were  collected  via  the 
Bedford  Workload  Rating  Scale  and  a  cockpit  controls  and  displays  usability 
questionnaire.  The  data  were  analyzed  to  determine  if  the  pilot  flying  the  aircraft 
(pilot  on  controls)  and  the  pilot  operating  the  mission  equipment  package  (MEP) 
experienced  tolerable  and  comparable  workload  levels  when  performing  35  crew 
member  tasks  (see  Appendix  A).  The  data  were  also  analyzed  to  determine 
whether  workload  levels  contribute  to  a  need  to  "battle  roster1"  Comanche  pilots. 

Results  indicate  that 

•  Workload  was  tolerable  for  the  pilots  when  they  performed  33  of  the 
35  crew  member  tasks. 

•  Workload  was  moderately  higher  for  pilots  when  they  operated  the 

MEP. 

•  There  is  no  compelling  need  to  prescribe  battle  rostering  for 
Comanche  pilots  because  (a)  workload  levels  were  tolerable  when  they 
performed  all  but  one  crew  member  task,  and  (b)  differences  in  workload  levels 
between  the  pilot  flying  the  aircraft  and  the  pilot  operating  the  MEP  were 
moderate. 


’Battle  rostering  is  defined  as  "the  designation  of  two  or  more  individuals  to  routinely  perform  as  a 
crew"  (Headquarters,  Department  of  the  Army,  1995). 
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ASSESSMENT  OF  CREW  WORKLOAD  FOR  THE  RAH-66  COMANCHE 
FORCE  DEVELOPMENT  EXPERIMENT  1 


1.  Introduction 


1.1  Purpose  and  Overview 

The  RAH-66  Comanche  Force  Development  Experiment  (FDE)  1  was  conducted 
from  June  6  to  29  2000  with  two  Comanche  portable  cockpit  (CPC)  devices  at  the 
Air  Maneuver  Battle  Laboratory,  Fort  Rucker,  Alabama.  It  was  the  first  of  six 
Force  Development  Test  and  Experimentation  (FDTE)  events  that  are  scheduled 
to  occur  from  Fiscal  Year  (FY)  2000  through  FY  2006.  The  purpose  of  FDE  1  was 
to  assess  whether  (a)  the  Comanche  pilot  and  crew  procedures  are  viable  for 
reconnaissance,  security,  and  attack  missions  in  a  variety  of  tactical  scenarios;  (b) 
workload  levels  are  evenly  and  appropriately  distributed  among  crew  members; 
and  (c)  crew  workload  levels  contribute  to  a  need  to  "battle  roster2"  Comanche 
pilots. 

The  Training  and  Doctrine  Command  (TRADOC)  System  Manager  -  Comanche 
(TSM-C)  was  the  sponsor  for  FDE  1.  The  TSM-C  requested  the  Human  Research 
and  Engineering  Directorate  of  the  U.S.  Army  Research  Laboratory  (ARL)  to 
assess  crew  workload  during  FDE  1.  ARL  collected  crew  workload  data  via  the 
Bedford  Workload  Rating  Scale  (see  Appendix  B)  and  a  cockpit  controls  and 
displays  usability  questionnaire  (Appendix  C).  The  data  were  analyzed  to 
determine  if  the  pilot  flying  the  aircraft  (pilot  on  controls)  and  the  pilot  operating 
the  mission  equipment  package  (MEP)  experienced  tolerable  and  comparable 
workload  levels  when  they  performed  35  crew  member  tasks  (see  Appendix  A). 
Additionally,  ARL  assessed  whether  workload  levels  contribute  to  a  need  to 
battle  roster  Comanche  pilots. 

1.2  Assessment  of  Crew  Workload 

A  common  definition  of  pilot  workload  is  "the  integrated  mental  and  physical 
effort  required  to  satisfy  the  perceived  demands  of  a  specified  flight  task" 
(Roscoe,  1985).  It  is  important  to  assess  pilot  workload  because  mission 
accomplishment  is  related  to  the  mental  and  physical  ability  of  the  crew  to 
effectively  perform  their  flight  and  mission  tasks.  If  one  or  both  pilots  experience 
high  workload  while  performing  flight  and  mission  tasks,  the  tasks  may  be 
performed  ineffectively  or  may  be  abandoned.  In  order  to  assess  whether  cockpit 
workload  is  evenly  and  appropriately  distributed  in  the  Comanche,  the  level  of 
workload  for  each  pilot  must  be  evaluated. 

battle  rostering  is  defined  as  "the  designation  of  two  or  more  individuals  to  routinely  perform  as  a 
crew"  (Headquarters,  Department  of  the  Army,  1995). 
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1.2.1  Bedford  Workload  Rating  Scale 

The  Bedford  Workload  Rating  Scale  has  been  used  extensively  by  the  military, 
civil,  and  commercial  aviation  communities  for  pilot  workload  estimation 
(Roscoe  &  Ellis,  1990).  It  requires  pilots  to  rate  the  level  of  workload  associated 
with  a  task,  based  on  the  amount  of  spare  capacity  they  feel  they  have  to  perform 
additional  tasks.  Spare  workload  capacity  is  an  important  commodity  for  pilots 
because  they  are  often  required  to  perform  several  tasks  concurrently.  For 
example,  pilots  often  perform  navigational  tasks,  monitor  radios,  and  assist  the 
pilot  on  controls  with  flight  tasks  (e.g.,  maintain  air  space  surveillance)  within 
the  same  time  interval.  Mission  performance  is  reduced  if  pilots  are  task 
saturated  and  have  little  or  no  spare  capacity  to  perform  other  tasks.  Design  of 
the  Comanche  pilot-vehicle  interface  should  help  ensure  that  pilots  can  maintain 
adequate  spare  workload  capacity  for  flight  and  mission  tasks. 

1.2.2  Battle  Rostering 

Battle  rostering  is  believed  to  improve  aircrew  workload  management  by 
enhancing  crew  coordination.  This  is  because  of  increased  familiarization  and 
confidence  between  pilots  who  routinely  train  together.  If  workload  levels  for 
Comanche  pilots  are  consistently  high  or  are  disproportionately  distributed, 
battle  rostering  may  help  them  manage  their  workload  more  effectively. 
However,  a  study  by  the  Army  Research  Institute  (Grubb,  Simon,  Leedom,  & 
Zeller,  1995)  has  questioned  the  benefits  of  battle  rostering.  The  study  identified 
potential  drawbacks,  including  crew  overconfidence,  increased  use  of  informal 
and  non-standard  procedures,  and  increased  reliance  on  implicit  coordination 
rather  than  on  explicit  coordination  behavior  between  pilots. 

1.3  FDE 1  Experimental  Procedure 

Pilots  received  4  weeks  of  intensive  training  before  the  FDE.  The  training 
consisted  of  classroom  instruction  and  "hands-on"  flight  experience  in  the  CPCs. 
The  pilots  flew  the  same  missions  (e.g.,  route  reconnaissance)  during  training 
that  they  later  flew  during  the  FDE.  The  mission  scenario  was  based  on 
battlefield  environments  simulating  those  depicted  in  the  Comanche  operational 
mode  summary  and  mission  profile.  The  scenario  was  conducted  in  four  phases 
(see  Table  1).  Each  successive  phase  increased  in  difficulty  in  order  to  challenge 
crew  workload.  The  pilots  performed  specific  planning  and  in-flight  tasks  (see 
Appendix  A)  during  each  mission.  Each  task  had  prescribed  conditions  and 
standards  that  both  crew  members  had  to  perform  to  help  ensure  mission 
accomplishment. 

Pilots  were  systematically  rotated  throughout  the  missions  so  that  every  pilot 
flew  equally  often  with  each  of  the  other  pilots.  For  all  missions,  the  pilot  flying 
the  aircraft  was  assigned  to  the  front  seat,  and  the  pilot  operating  the  MEP  was 
assigned  to  the  back  seat.  Pilots  alternated  between  the  front  seat  and  back  seat 
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positions  during  each  phase.  A  total  of  30  missions  was  conducted  during  the 
FDE. 


Table  1.  Phases  of  Force  Development  Experiment  1 


Phase 

Mission  description 

Mission  objectives 

1 

Conduct  route  reconnaissance 
Conduct  point  reconnaissance 
(bridges) 

Fly  to  holding  area 

Navigation,  basic  mission  equipment 
manipulation,  and  aircraft  control. 
Arrive  at  holding  area  undetected. 

2 

Conduct  route  reconnaissance 
Conduct  area  reconnaissance 
Provide  security  (screen) 

Engage  enemy  with  artillery 

Navigation,  advanced  mission 
equipment  manipulation,  digital 
communications,  call  for  fire 

3 

Conduct  security  operations 
(screen) 

Conduct  deep  reconnaissance 
Attack  theater  ballistic  missiles 
React  to  mission  change 

React  to  changes  in  weather 

All  the  above,  react  to  mission 
changes,  and  execute  procedures  for 
inadvertent  entry  into  instrument 
meteorological  conditions 

4a 

ax  T/^vnn'n. 

Conduct  zone  reconnaissance 
React  to  mission  change 

React  to  changes  in  weather 

All  the  above 

■’NOTE:  Crews  were  battle  rostered 


1.4  System  Description 

1.4.1  Comanche  Portable  Cockpit  (CPC) 

The  CPC  (see  Figure  1)  consists  of  two  (non-motion)  cockpits  arranged  in  a 
tandem  configuration  and  mounted  in  a  transport  trailer.  The  front  and  rear 
cockpits  are  identical,  allowing  each  pilot  to  perform  all  crew  member  tasks.  The 
CPC  contains  the  hardware,  MEP,  and  software  that  simulate  the  controls,  flight 
characteristics,  and  functionality  of  the  Comanche  aircraft.  The  primary  cockpit 
controls  and  displays  are  the  system  management  display  (SMD),  tactical 
situation  display  (TSD),  cockpit  interactive  keyboard  (CIK),  side  arm  controller 
(SAC),  collective,  and  the  Kaiser  ProView  50™  head-mounted  display  (HMD). 


Two  CPCs  were  used  during  FDE  1.  They  were  programmed  with  the  latest 
version  of  flight  and  mission  software  in  January  2000.  The  major  differences 
between  the  CPCs  that  were  used  for  the  FDE  and  the  Comanche  aircraft  are 
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•  The  Kaiser  ProView  50™  HMD  was  used  in  lieu  of  the  Helmet 
Integrated  Display  and  Sighting  Subsystem  (HIDSS).  The  HIDSS  is  being 
developed  and  will  be  the  production  HMD  used  by  Comanche  pilots. 

•  The  cockpit  geometry  of  the  CPCs  was  similar  but  not  identical  to  that 
of  the  Comanche  aircraft. 

•  CPCs  were  not  equipped  with  a  motion  system. 


1.4.2  System  Management  Display  (SMD)  and  Tactical  Situation  Display  (TSD) 

The  SMD  is  a  multi-function  color  display.  In  one  mode,  it  provides  sensor 
imagery  from  the  target  acquisition  system  (TAS).  In  other  modes,  it  provides 
aircraft  subsystem  control  and  status  information.  The  TSD  is  also  a  multi¬ 
function  display.  It  provides  a  color  map  and  navigational  information, 
including  the  location  of  threat  and  friendly  forces.  The  SMD  and  TSD  each  have 
a  bezel  incorporating  12  dedicated  switches  (called  mode  select  keys)  in  two 
horizontal  rows  above  and  below  each  of  the  displays.  The  six  mode  select  keys 
on  the  upper  bezel  of  the  SMD  are  used  to  select  communication  functions,  while 
the  six  mode  select  keys  on  the  lower  bezel  allow  selection  of  the  main  menu  of 
the  SMD  or  aircraft  and  mission  subsystems.  The  six  mode  select  keys  on  the 
upper  row  of  the  TSD  bezel  are  used  to  select  HMD  functions.  The  six  mode 
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select  keys  on  the  lower  TSD  bezel  allow  manipulation  of  map  modes  and 
display  characteristics.  Switches  in  the  corners  of  the  bezels  are  used  to  adjust 
screen  brightness,  symbol  brightness,  and  contrast.  There  are  10  switches  in  two 
vertical  rows  on  the  right  and  left  of  the  SMD  and  TSD.  The  function  and  use  of 
these  keys  vary,  depending  on  the  system  or  subsystem  being  viewed. 

1.4.3  Left  and  Right  Multipurpose  Displays  (MPDs) 

The  left  MPD  (LMPD)  is  located  outboard  of  the  SMD,  and  the  right  MPD 
(RMPD)  is  located  outboard  of  the  TSD.  The  lower  segment  of  the  LMPD 
contains  line  address  keys,  and  the  upper  segment  presents  the  status  of 
selections  made  from  the  tactical  interactive  annunciator  panel.  The  lower 
segment  of  the  RMPD  provides  selective  monitoring  of  vehicle  subsystems  and 
displays  the  current  settings  (frequency,  channel  preset,  transmitter,  and 
ciphony)  of  the  communication  radios.  The  upper  segment  of  the  RMPD  screen 
provides  information  about  the  operational  status  and  modes  of  the  weapon 
system  and  mission  equipment. 

1.4.4  Collective  and  Sidearm  Controller  (SAC) 

The  collective  is  located  to  the  left  of  the  crew  member's  seat,  and  the  SAC  is 
located  on  the  right  armrest.  The  SAC  allows  pilots  to  control  the  pitch,  roll,  and 
yaw  of  the  aircraft.  It  also  allows  10%  authority  vertical  input.  The  collective 
permits  full  authority  vertical  input.  Both  the  collective  and  SAC  grips  contain 
switches  that  allow  hands-on  control  of  critical  flight  and  mission  functions.  The 
right  armrest  also  contains  switches  that  control  mission  functions.  The 
collective,  SAC,  and  right  armrest  switches  are  listed  in  Appendix  D. 

1.4.5  Cockpit  Interactive  Keyboard  (CIK) 

The  CIK  enables  crew  members  to  enter  data  into  the  computer  system.  The  data 
include  radio  frequencies,  coordinates,  targets,  and  text  messages. 

1.4.6  Helmet-Mounted  Display  (HMD) 

The  Kaiser  ProView  50™  (see  Figure  2)  was  the  HMD  used  by  pilots  during  FDE 
1.  It  had  two  liquid  crystal  displays  with  28°  (V)  by  49°  (H)  field  of  view  (FOV) 
(25%  binocular  overlap),  1024  by  768  resolution,  inter-pupillary  distance 
adjustment,  eye  relief  adjustment,  adjustable  headband  and  strap,  an  electronic 
control  unit,  and  a  Polhemus™  head-tracking  sensor.  The  weight  of  the  HMD 
was  1.3  pounds.  The  HMD  displayed  monochrome  symbology  overlaid  on  the 
synthetic  visual  scene.  When  used  in  the  night  vision  pilotage  system  (NVPS) 
mode,  the  HMD  displayed  the  forward-looking  infrared  (FLIR)  scene  overlaid  by 
the  monochrome  symbology.  The  HMD  also  displayed  an  image  intensification 
(I2)  scene  overlaid  by  the  monochrome  symbology.  Pilots  selected  which  scene 
(FLIR  or  I2)  was  displayed  on  the  HMD.  A  headset  was  placed  over  the  HMD  to 
provide  the  pilots  with  the  capability  for  radio  and  inter-cockpit  communication. 


7 


Figure  2.  Kaiser  ProView  50™. 


2.  Method 


2.1  Participants 

Participants  were  six  male  Army  pilots  from  the  following  units:  8-101st 
Aviation  Regiment,  Fort  Campbell,  Kentucky,  l-82nd  Aviation  Regiment,  Fort 
Bragg,  North  Carolina,  2-17th  Cavalry  Squadron,  Fort  Campbell  (two  pilots),  3- 
229th  Aviation  Regiment,  Fort  Bragg,  and  l-3rd  Aviation  Regiment,  Fort  Stewart, 
Georgia.  Three  participants  were  OH-58D  pilots  who  held  the  rank  of  CW2.  The 
other  three  participants  were  AH-64A  pilots  who  held  the  rank  of  CW3.  They 
represented  a  group  of  moderately  experienced  pilots  with  total  flight  hours  that 
ranged  from  785  hours  to  2,170  hours  in  Army  aircraft.  None  of  the  participants 
had  previous  experience  flying  a  Comanche  simulator  or  aircraft.  The  relevant 
demographic  characteristics  of  the  pilots  are  listed  in  Table  2. 


Table  2.  Demographic  Characteristics  of  Pilots 
(N  =  6) 


Summary  of 

demographic 

characteristics 

Age 

(yrs) 

Flight  hours 
in  primary 
aircraft 

Total  flight 
hours  in 

Army  aircraft 

Flight  hours 
with  night 
vision  devices 

Average 

34 

910 

1292 

480 

Median 

34 

800 

1180 

435 

Range 

26-41 

620-1500 

785-2170 

352-707 
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2.2  Data  Collection 


Two  questionnaires  were  developed  in  order  to  assess  workload  for  the  crew 
member  tasks  listed  in  Appendix  A.  One  of  the  questionnaires  required  the  pilots 
to  rate  their  perceived  level  of  workload  for  each  crew  member  task  via  the 
Bedford  Workload  Rating  Scale.  The  other  questionnaire  required  pilots  to  rate 
whether  (a)  usability  features  of  the  cockpit  controls  and  displays  impacted  their 
workload  levels  during  the  mission,  and  (b)  workload  was  higher  for  the  pilot 
flying  the  aircraft  (front  seat)  or  the  pilot  operating  the  MEP  (back  seat).  The 
purpose  of  this  questionnaire  was  to  highlight  any  usability  problems  that 
contributed  to  excessive  workload  during  the  mission  and  to  help  determine  if 
workload  was  evenly  distributed  between  pilots. 

The  questionnaires  were  developed  in  accordance  with  published  guidelines  for 
proper  format  and  content  (Babbitt  &  Nystrom,  1989).  A  brief  pre-test  was 
conducted  to  refine  the  questionnaires  and  to  ensure  that  they  could  be  easily 
understood  and  completed  by  pilots.  Before  experimentation,  the  FDE  pilots 
were  briefed  about  the  purpose  of  the  workload  assessment.  They  also  received 
instruction  about  completing  both  questionnaires. 

The  pilots  provided  workload  ratings  for  each  applicable  crew  member  task 
immediately  after  every  mission.  They  completed  the  cockpit  controls  and 
displays  usability  questionnaire  at  the  end  of  each  phase.  Questionnaire  results 
were  clarified  with  information  obtained  during  post-flight  discussions  with 
pilots. 

2.3  Data  Analysis 

Pilot  responses  to  the  workload  rating  questionnaire  and  cockpit  controls  and 
displays  usability  questionnaire  were  analyzed  with  averages  and  percentages. 
In  addition,  the  sign  test  was  used  to  identify  any  statistically  significant 
differences  in  workload  ratings  between  the  pilot  on  controls  and  the  pilot 
operating  the  MEP.  The  chi-square  goodness-of-fit  and  binomial  tests  were  used 
to  identify  any  statistically  significant  response  trends  to  questionnaire  items 
regarding  whether  the  usability  characteristics  of  the  cockpit  controls  and 
displays  impacted  workload.  Statistically  significant  trends  indicate  that  the 
responses  provided  by  the  pilots  to  the  questionnaire  items  were  not  random  but 
were  attributable  to  factors  such  as  strong  favorable  or  unfavorable  opinions 
about  usability  characteristics  of  the  cockpit  controls  and  displays.  Because  of  the 
small  number  of  pilots  who  participated  in  the  FDE,  probability  values  were 
computed  via  Fisher's  Exact  Test. 

Because  the  pilots  had  limited  experience  with  the  CPCs  before  experimentation 
began,  correlation  coefficients  were  computed  to  identify  any  significant  changes 
in  their  responses  to  the  controls  and  displays  usability  questionnaire  over  the 
course  of  the  FDE.  This  was  done  to  help  determine  if  increased  experience  with 
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the  system  significantly  affected  pilot  responses  to  questionnaire  items.  For 
example,  pilots  might  rate  a  component  as  much  easier  to  use  toward  the  end  of 
the  FDE  because  of  increased  experience  with  using  it  during  previous  missions. 
Significant  changes  in  their  responses  might  indicate  that  (a)  pilots  would  have 
benefited  from  additional  training,  and  (b)  the  additional  training  would  have 
allowed  the  pilots  to  provide  more  valid  responses  during  the  early  phases  of  the 
FDE. 


2.4  Limitations  of  Workload  Assessment 

Limitations  included  the  small  sample  size  of  pilots  (N  =  6)  who  participated  in 
the  FDE,  their  limited  experience  (2  months)  with  the  CPCs,  and  the  lack  of  100% 
fidelity  between  the  CPCs  and  the  Comanche  aircraft.  Information  and  data 
listed  in  the  Results  and  Summary  sections  of  this  report  should  be  interpreted 
on  the  basis  of  these  limitations. 


3.  Results 


3.1  Workload  When  Pilots  Performed  Crew  Member  Tasks 

As  summarized  in  Table  3  and  Appendix  E,  pilots  reported  that  workload  was 
tolerable  when  they  performed  all  front  and  back  seat  tasks  except  "maintaining 
air  space  surveillance".  Pilots  in  the  back  seat  could  not  maintain  air  space 
surveillance  at  night  because  they  had  no  night  vision  device  (FLIR  or  night 
vision  goggles)  to  see  outside  the  cockpit  while  they  conducted  a  scan  with  the 
TAS. 


3.2  Spare  Workload  Capacity  When  Pilots  Performed  Crew  Member 
Tasks 

Pilots  reported  that  they  experienced  no  reduction  in  spare  workload  capacity 
while  they  performed  24  (69%)  crew  member  tasks  in  the  front  seat  and  17  (63%) 
tasks  in  the  back  seat.  When  flying  the  aircraft  from  the  front  seat,  pilots  reported 
that  they  experienced  "insufficient  spare  workload  capacity  for  easy  attention  to 
additional  tasks"  while  they  performed  11  (31%)  crew  member  tasks.  This  is 
understandable  for  8  of  the  11  tasks  because  they  are  inherently  workload¬ 
intensive  events  because  pilots  are  (a)  attempting  to  engage  or  avoid  the  threat  or 
(b)  responding  to  an  emergency  or  unexpected  event.  The  other  three  tasks 
(digital  communications,  data  entry  procedures,  and  data  management 
operations)  received  this  rating  primarily  because  of  usability  problems  with  the 
CIK  and  the  TSD  (see  Table  4).  Pilots  reported  that  entering  data  on  the  CIK  was 
time  consuming  and  required  excessive  steps.  Specific  items  that  were  reported 
as  being  difficult  to  enter  included  radio  frequencies,  targets,  and  coordinates. 
The  slow  processing  speed  of  the  CIK  was  also  cited  as  a  usability  problem. 
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Pilots  reported  that  labeling  targets  on  the  TSD  was  time  consuming  because  of 
the  excessive  number  of  steps  required  to  perform  the  task. 

Table  3.  Summary  of  Workload  Ratings 


Pilot  on  controls  (front  seat) 


Pilot  operating  the  MEP  (back  seat) 


Workload  was  tolerable  when  pilots 
performed  all  tasks 

Pilots  experienced  no  reduction  in  spare 
workload  capacity  when  performing  24 
of  35  tasks  (69%) 

Pilots  experienced  "insufficient  spare  work¬ 
load  capacity  for  easy  attention  to  additional 
flight  and  mission  tasks"  when  performing 

•  Evasive  maneuvers 

•  Firing  techniques 

•  Inadvertent  IMC  procedures 

•  Data  management  operations 

•  Target  engagement  with  AWS 

•  Target  engagement  with  PTWS 

•  Actions  on  contact 

•  Firing  position  operations 

•  Emergency  procedures 

•  Data  entry  procedures 

•  Digital  communications 


Workload  was  tolerable  when  pilots 
performed  all  tasks  except  "maintaining 
air  space  surveillance" 

Pilots  experienced  no  reduction  in 
spare  workload  capacity  when 
performing  17  of  27  tasks  (63%) 

Pilots  experienced  "insufficient  spare  work¬ 
load  capacity  for  easy  attention  to  additional 
flight  and  mission  tasks"  when  performing 

•  Radio  communications 

•  TSD  operations 

•  Target  engagement  with  AWS 

•  Identification  of  major  U.S.-allied  equip¬ 

ment  and  major  threat  equipment 

•  Operation  of  the  NVPS 

•  Inadvertent  IMC  procedures 

Pilots  experienced  "reduced  spare  workload 
capacity."  Additional  tasks  could  not  be 
given  the  desired  amount  of  attention  when 
performing: 

•  Data  management  operations 

Pilots  experienced  "little  spare  workload 
capacity;  their  level  of  effort  allowed  little 
attention  to  additional  tasks"  when 
performing 

•  Digital  communications 

•  Data  entry  procedures 

Pilots  could  not  perform  the  following  task  at 
night  because  they  did  not  have  a  night 
vision  device  to  see  outside  the  cockpit 
while  conducting  a  scan  with  the  TAS 

•  Maintaining  air  space  surveillance 


IMC  =  instrumented  meteorological  conditions 
AWS  =  area  weapon  system 
PTWS  =  point  target  weapon  system 
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Table  4.  Impact  of  Usability  Characteristics  of  Cockpit 
Controls  and  Displays  on  Workload 


CIK 

MPDs 

SMD 

TSD 

Steps  to  accomplish  task 

Pilots  reported  that 
using  the  CIK  was 
significantly  time 
consuming  for  most 
missions  because 
steps  required  to 
accomplish  functions 
were  excessive  and  not 
logical  or  consistent. 
Pilots  reported  that 
the  CIK  should 
be  modified  to 
decrease  workload. 

Pilots  reported  that 
using  the  MPDs  was 
not  significantly  time 
consuming  (for  most 
missions)  because  of 
the  steps  required  to 
complete  a  function. 
Pilots  reported  that 
some  modifications 
should  be  made  in 
the  MPDs  to  help 
decrease  workload. 

Pilots  reported  that 
using  the  SMD  was 
not  significantly  time 
consuming  (for  most 
missions)  because  of 
the  steps  required  to 
complete  a  function. 
Pilots  reported  that 
some  modifications 
should  be  made  in 
the  SMD  to  help 
decrease  workload. 

Pilots  reported  that  using 
the  TSD  was  significantly 
time  consuming  for  50% 
of  missions  because  steps 
required  to  accomplish 
functions  were  excessive 
and  not  logical  or  consistent. 
Pilots  reported  that  modifi¬ 
cations  should  be  made  in 
the  TSD  to  decrease  workload. 

Menu  navigation 

Not  applicable 

Pilots  were  able  to 
quickly  navigate 
through  the  menu 
screens  for  96% 
of  missions. 

Pilots  were  able  to 
quickly  navigate 
through  the  menu 
screens  for  91% 
of  missions. 

Pilots  were  able  to 
quickly  navigate 
through  the  menu 
screens  for  71% 
of  missions. 

Not  applicable 

Pilots  seldom  or 
never  had  trouble 
remembering  where 
they  were  in  the 
menu  system  for 
all  missions. 

Pilots  seldom  or 
never  had  trouble 
remembering  where 
they  were  in  the 
menu  system  for 
all  missions. 

Pilots  seldom  or 
never  had  trouble 
remembering  where 
they  were  in  the 
menu  system  for 
all  missions. 

Symbology 

Not  applicable 

Not  applicable 

Pilots  reported  that 
they  could  quickly 
and  easily  understand 
symbology  displayed 
on  the  SMD  during 
all  missions. 

Pilots  reported  that 
they  could  quickly 
and  easily  understand 
symbology  displayed 
on  the  TSD  for  65%  of 
missions.  Pilots  reported 
that  some  modifications 
of  the  symbology 
should  be  made  to 
decrease  workload. 

When  operating  the  MEP  in  the  back  seat,  pilots  reported  that  they  experienced 
"insufficient  spare  workload  capacity  for  easy  attention  to  additional  tasks"  while 
they  performed  six  (22%)  crew  member  tasks.  This  is  understandable  for  four  of 
the  six  tasks  because  they  are  inherently  workload-intensive  events  because 
pilots  are  (a)  attempting  to  identify,  engage,  or  avoid  the  threat  or  (b)  responding 
to  an  unexpected  event  (inadvertent  IMC).  The  other  two  tasks  (radio 
communications  and  TSD  operations)  received  this  rating  primarily  because  of 
usability  problems  with  the  CIK  and  the  TSD.  The  usability  problems  are  the 
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same  as  those  identified  by  the  pilots  when  they  flew  the  aircraft  from  the  front 
seat  (see  previous  paragraph).  Pilots  reported  that  they  experienced  "reduced 
spare  workload  capacity"  while  they  performed  data  management  operations 
and  "little  spare  workload  capacity"  while  they  performed  digital 
communications  and  data  entry  procedures.  Again,  pilots  cited  usability 
problems  with  the  CIK  and  TSD  as  the  primary  reasons  for  assigning  these 
ratings  to  the  tasks.  Finally,  pilots  could  not  maintain  air  space  surveillance  at 
night  because  they  did  not  have  a  night  vision  device  to  see  outside  the  cockpit 
while  they  conducted  a  scan  with  the  TAS. 

3.3  Comparative  Workload  Levels  Between  Front  Seat  and  Back  Seat 

No  statistically  significant  differences  in  workload  ratings  were  provided  by 
pilots  when  they  flew  the  aircraft  versus  when  they  operated  the  MEP.  However, 
when  asked  to  rate  whether  overall  workload  was  higher  for  the  front  or  back 
seat  for  the  missions  they  performed,  pilots  reported  that  workload  was 
somewhat  higher  (54%)  or  much  higher  (22%)  for  the  back  seat  compared  to  the 
front  seat.  The  primary  reason  given  by  the  pilots  was  that  operating  the  MEP 
required  the  crew  member  to  perform  more  tasks  in  the  same  time  interval  than 
the  crew  member  who  was  flying  the  aircraft.  Also,  pilots  reported  during  post¬ 
flight  discussions  that  usability  problems  with  the  CIK,  TSD,  and  "slew"  hook- 
FOV  switches  on  the  collective  grip  contributed  to  higher  workload  for  the  pilot 
in  the  back  seat.  The  pilot  operating  the  MEP  uses  the  CIK,  TSD,  and  FOV-slew 
switches  more  often  during  the  mission  than  does  the  pilot  flying  the  aircraft. 

3.4  Pilot  Responses  to  Controls  and  Displays  Usability  Questionnaire 

Increased  experience  with  the  CPC  over  the  course  of  the  FDE  did  not  affect  pilot 
responses  to  the  controls  and  displays  usability  questionnaire  items.  No 
significant  correlation  coefficients  were  obtained  when  pilot  responses  were 
analyzed.  The  ratings  provided  by  the  pilots  were  consistent  between  each  phase 
of  the  FDE.  This  indicates  that  the  training  they  received  before  the  FDE  was 
sufficient  for  them  to  identify  the  usability  characteristics  of  cockpit  controls  and 
displays  that  contributed  to  increased  workload.  Discussions  with  pilots  also 
indicated  that  they  were  able  to  easily  identify  (early  in  training)  the  usability 
characteristics  of  the  controls  and  displays  that  contributed  to  higher  workload. 

3.5  HMD  Symbology 

Pilots  reported  that  it  was  difficult  to  quickly  and  easily  understand  the  heading 
tape  displayed  in  the  HMD  when  they  flew  the  aircraft  or  operated  the  MEP. 
They  unanimously  requested  that  the  heading  tape  be  stabilized  and  not  move 
when  the  pilot  moves  his  head.  The  pilots  also  requested  that  pitch  ladder  and 
bank  angle  symbols  be  added  to  the  HMD  symbology  set. 
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4.  Summary 


4.1  Cockpit  Workload 

Results  indicate  that  (a)  workload  was  tolerable  for  pilots  when  they  performed 
all  but  one  crew  member  task;  (b)  pilots  experienced  no  reduction  in  spare 
workload  capacity  for  69%  of  tasks  when  they  flew  the  aircraft  and  for  61%  of 
tasks  when  they  operated  the  MEP;  and  (c)  workload  was  moderately  higher  for 
pilots  when  they  operated  the  MEP  versus  when  they  flew  the  aircraft.  Following 
is  a  brief  summary  of  the  results: 

•  Pilots  experienced  tolerable  workload  levels  for  all  tasks  when  they 
flew  the  aircraft.  They  experienced  tolerable  workload  levels  for  all  but  one  task 
when  they  operated  the  MEP.  Workload  was  not  tolerable  for  the  task  of 
"maintaining  air  space  surveillance"  at  night  for  the  pilot  operating  the  MEP. 
This  was  because  there  was  no  night  vision  device  for  seeing  outside  the  cockpit 
when  the  pilots  conducted  a  scan  with  the  TAS. 

•  Pilots  experienced  no  reduction  in  spare  workload  capacity  for  69%  of 
tasks  when  they  flew  the  aircraft  and  for  61%  of  tasks  when  they  operated  the 
MEP.  For  tasks  in  which  a  reduction  in  spare  workload  capacity  was 
experienced,  the  primary  reasons  were 

•  The  tasks  were  inherently  workload  intensive  because  pilots  were 
identifying,  engaging,  or  avoiding  a  threat  or  responding  to  an  emergency  or 
unexpected  event. 

•  Usability  problems  with  the  CIK,  TSD,  and  FOV-slew  switches  on 
the  collective. 

•  Workload  levels  were  higher  for  pilots  when  they  operated  the  MEP. 
The  primary  reasons  were 

•  Operating  the  MEP  required  pilots  to  perform  more  tasks  during 
the  mission  than  when  they  flew  the  aircraft. 

•  Usability  problems  with  the  CIK,  TSD,  and  FOV-slew  switches  on 
the  collective  increased  workload  for  pilots  when  they  operated  the  MEP.  The 
pilot  operating  the  MEP  uses  the  CIK,  TSD,  and  FOV-slew  switches  more  often 
during  the  mission  than  does  the  pilot  flying  the  aircraft. 
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•  Pilots  unanimously  requested  that  the  heading  tape  displayed  in  the 
HMD  be  stabilized  and  not  move  when  the  pilot  moves  his  head.  They  also 
requested  that  pitch  ladder  and  bank  angle  symbols  be  added  to  the  HMD 
symbology  set. 

4.2  Battle  Rostering  of  Comanche  Pilots 

Results  indicate  that  there  is  no  compelling  need  to  prescribe  battle  rostering 
because  of  high  or  disproportionately  distributed  workload  between  Comanche 
pilots.  This  is  because  (as  summarized  on  the  previous  page) 

•  Workload  was  tolerable  for  all  but  one  of  the  crew  member  tasks 
performed  by  pilots. 

•  Pilots  experienced  no  reduction  in  spare  workload  capacity  for  69%  of 
tasks  when  they  flew  the  aircraft  and  for  61%  of  tasks  when  they  operated  the 
MEP. 


•  Only  moderate  differences  in  workload  levels  were  reported  between 
pilots  when  they  flew  the  aircraft  versus  when  they  operated  the  MEP. 


5.  Recommendations 


Data  obtained  during  FDE  1  provide  useful  insights  about  workload  levels  that 
Comanche  pilots  may  experience  when  performing  crew  member  tasks.  The  data 
also  identified  usability  limitations  with  the  CIK,  TSD,  slew  hook  and  FOV 
switches  on  the  collective,  and  symbology  on  the  HMD  (see  Appendix  C).  These 
usability  limitations  should  be  addressed  and  resolved  as  soon  as  possible. 
Additionally,  the  pilot  operating  the  MEP  should  be  provided  with  a  night  vision 
device  to  see  outside  the  cockpit  at  night  when  he  conducts  a  scan  with  the  TAS. 

The  scope  of  FDE  1  did  not  allow  a  comprehensive  evaluation  of  workload  for 
Comanche  pilots.  To  fully  assess  the  design  of  the  Comanche,  future  FDTE 
events  should  include  comprehensive  evaluations  of  pilot  workload.  Other 
(related)  performance  measures  such  as  crew  coordination,  situational 
awareness,  and  decision  making  should  be  evaluated  to  help  assess  workload 
and  fully  develop  the  pilot-vehicle  interface. 
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APPENDIX  A 


FDE  1  CREW  MEMBER  TASKS 
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FDE 1  Crew  Member  Tasks 


Task  1035  -  Perform  Before 
Flight  Checks 

Task  1162  -  Perform 
Actions  On  Contact 

Task  1442  -  Perform  HIDSS 
Operations 

Task  1042  -  Maintain 
Airspace  Surveillance 

Task  1168  -  Select  Landing 
Zone/Holding  Area 

Task  1448  -  Perform 
EOTADS  Sensor 
Operations 

Task  1100  -  Perform  Radio 
Communications 

Task  1173  -  Perform  VMC 
Approach  and  Roll-on 
Landing 

Task  1449  -  Perform  Digital 
Communications 

Task  1107  -  Perform 
Hovering  Flight 

Task  1230  -  Perform 
Inadvertent  IMC 
Procedures 

Task  1454  -  Perform  Data 
Entry  Procedures 

Task  1109  -  Perform  VMC 
Takeoff 

Task  1300  -  Perform 
Emergency  Procedures 

Task  1455  -  Perform  Data 
Management  Operations 

Task  1117  -  Perform  VMC 
Flight  Maneuvers 

Task  1410  -  Perform  TSD 
Operations 

Task  1458  -  Engage  Target 
With  PTWS 

Task  1127  -  Perform 
Electronically  Aided 
Navigation 

Task  1414  -  Perform  Target 
Handover 

Task  1464  -  Engage  Target 
With  The  AWS  Turreted 
Gun 

Task  1136  -  Perform 
Terrain  Flight  Navigation 

Task  1422  -  Perform  Firing 
Techniques 

Task  1500  -  Perform  Aerial 
Observation 

Task  1146  -  Perform 
Terrain  Flight 

Task  1426  -  Perform  Firing 
Position  Operations 

Task  1805  -  Identify  Major 
US- Allied  Equipment  and 
Threat  Equipment 

Task  1148  -  Perform 
Terrain  Flight  Deceleration 

Task  1427  -  Select  A 
Combat  Position 

Task  1823  -  Operate 
Aircraft  Survivability 
Equipment 

Task  1151  -  Perform 
Masking  And  Unmasking 

Task  1428  -  Select 
Appropriate  Weapon 
System 

Task  1837  -  Operate  Night 
Vision  Pilotage  System 

Task  1153  -  Perform 
Evasive  Maneuvers 

Task  1440  -  Perform  HIDSS 
Boresight 

VMC  =  visual  meteorological  conditions 

EOT  ADS  =  electro-optical  target  acquisition  designation  system 
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APPENDIX  B 


BEDFORD  WORKLOAD  RATING  SCALE 
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Workload  Description  "Rating" 


Workload  insignificant 

MM 

Workload  low 

2 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

*  Was  workload 
satisfactory 
without  reduction  in 
spare  (workload)  capacity? 


[  YES  | 


Was  workload  tolerable 
for  the  task? 


Was  it  possible  to 
complete  the  task? 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 


Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 


Little  spare  capacity:  level  of  effort 
allows  little  attention  to  additional 
tasks 


Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 


Very  high  workload  with  almost  no 
spare  capacity.  Difficulty  in 
maintaining  level  of  effort 


Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 


Task  abandoned.  Pilot  unable  to 

10 

apply  sufficient  effort 

Pilot  Decisions 
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APPENDIX  C 


SUMMARY  OF  PILOT  RESPONSES  ABOUT  THE  USABILITY 
CHARACTERISTICS  OF  THE  CREW  STATION 
CONTROLS  AND  DISPLAYS 
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SUMMARY  OF  PILOT  RESPONSES  REGARDING  USABILITY 
CHARACTERISTICS  OF  CREW  STATION  CONTROLS  AND  DISPLAYS 


Any  CIK  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to 
Excessive  Number  of  Steps**? 


Any  MPD  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to 
Excessive  Number  of  Steps? 


0%  20%  40%  60%  80%  100% 


3 

BMP  70i 

0 

0%  20%  40%  60%  80%  100% 


Pilot  Comments: 


Pilot  Comments : 


Problems  when  using  the  CIK: 

•  CIK  is  too  cumbersome  (4  comments). 

•  CIK  is  too  cumbersome  when  data  are 
entered. 

•  The  CIK  is  cumbersome  and  slow. 

•  Cumbersome-  would  like  voice  interaction 

•  Excessive  number  of  steps  for  CIK  use. 

•  All  (CIK,  MPDs,  SMD,  and  TSD)  required 
an  excessive  number  of  steps. 

•  Too  many  steps  to  enter  anything  in  the 
CIK. 

•  Location  and  interface  difficult. 

•  Need  quicker  method  of  entering  manual 
frequencies  and  waypoints  for  direct  to. 

•  CIK  should  be  enabled  for  direct  waypoint 
entry  by  8-digit  grid. 

•  Manual  entry  at  grid  for  FLT  plan,  direct 
waypoints  (should  be  more  interactive.). 

•  Trying  to  make  sentences  takes  too  long. 

•  Keyboard  needs  to  be  able  to  be  interactive 
with  more  functions,  such  as  entering  a 
direct  waypoint,  or  a  manual  radio 
frequency. 

•  Difficult  to  interact  with.  Would  like  a 
manual  selection  to  enter  commo,  freqs,  or 
grid  locations.  Very  slow  and  not  user 
friendly  when  interacting  with  commo  and 
NAV  package. 

•  Changing  radio  frequencies,  it  would  be 
nice  to  have  a  control  head.  It  would  be  a 
lot  easier  to  change  a  manual  frequency. 

•  Key  positioning  requires  "hunt  and  peck." 

•  It  is  too  slow. 


**Significant  at  a  .01 


RMPD-  Fuel  quantity  indicator  is  very 
hard  to  focus  on.  When  checking  fuel,  I 
find  myself  "inside  the  cockpit"  far  too 
long. 

Fuel  display. 

RMPD  is  cluttered  and  is  very  hard  to 
disseminate  info. 

When  selecting  a  radio,  I  spent  too  much 
time  looking  down  at  RFD.  Difficult  to  see 
the  display  and  forced  to  look  at  it  when 
selecting  radio. 

Multi-level  menus  are  sometimes  too  deep. 
When  selecting  a  hard  bezel  should  always 
go  to  top  TMI.  Recognition  of  what  level 
you  are  in  requires  time. 

Visual  indication  of  20  mm  deployed 
(picture). 
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Any  SMD  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to 
Excessive  Number  of  Steps? 


Any  TSD  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to 
Excessive  Number  of  Steps? 


Pilot  Comments: 

•  Any  hard  bezel  selection  should  kick  you 
out  of  the  slaved  function.  (4  comments) 

•  Windows  and  digital  communications 

•  Communication  pages 

•  Windows. 

•  TAS  manual/use. 

•  TAS  sensitive  to  aircraft  in  main  slew 
mode. 


Pilot  Comments: 

Problems  with  labeling  of  targets: 

•  TGT  labeling. 

•  Labeling  targets  is  time  consuming. 
Suggest  a  selectable  bar  on  TSD. 

•  Labeling  function  too  time  consuming  and 
cumbersome/  windows. 

•  Labeling  takes  too  long. 

•  Labels. 

•  Label  function  excessive. 

•  Labeling  targets  is  very  time  consuming. 

•  Labeling  of  TGTS  should  be  a  selection  on 
LMPD.  Current  method  is  time  consuming 
and  awkward. 

Other  comments: 

•  Icons  should  be  scaled  to  map.  (2 
comments) 

•  Threat  ID  classification.  Right  slew  hook 
and  design  should  be  moved  to  a  time 
menu  item  on  LMPD. 

•  MAP  is  cluttered. 

•  Hook  function  and  picking  between  two 
targets  that  are  very  close  together. 
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Any  CIK  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to  Steps 
That  Were  Not  Logical  or  Consistent? 


Any  MPD  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to  Steps 
That  Were  Not  Logical  or  Consistent?**? 


— 

\  64% 

. 7 

0%  20%  40%  60%  80%  100% 


0%  20%  40%  60%  80%  100% 


Pilot  Comments: 


Pilot  Comments: 


Problems  when  using  the  CIK: 


Too  cumbersome.  (2  comments) 

All  CIK  functions.  (2  comments) 

The  keyboard  needs  to  be  designed  with  a 
"PC  style  layout  (2  comments) 

Too  many  steps. 

Too  many  steps  for  use. 

This  CIK  takes  too  long  to  utilize. 

Too  slow. 

The  CIK  takes  too  long  to  input  data. 
Manual  entry  for  both  frequencies  and 
waypoints. 

Should  always  be  able  to  activate  keyboard 
for  grids-spot  reports 


The  direct  to  function  in  NAV-CUR  is  not 
in  logical  sequence  when  using  in 
conjunction  with  locate. 

Multiple  level  menus. 

There  were  too  many  items  that  are  split 
between  the  TIMI  menu  structure  and  hard 
bezels  on  the  MPD's. 

When  selecting  hard  bezel,  should  always 
go  to  top  TMI. 


Any  SMD  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to  Steps 
That  Were  Not  Logical  or  Consistent?**? 


Any  TSD  Functions  That  Were  Significantly 
Time  Consuming  to  Accomplish  Due  to  Steps 
That  Were  Not  Logical  or  Consistent? 


0%  20%  40%  60%  80%  100% 


0%  20%  40%  60%  80%  100% 


Pilot  Comments: 


Pilot  Comments: 


Windows.  (2  comments) 

Using  any  function  on  SMD,  you  lose  sight 
of  target  in  TAS. 

Communication  systems. 


•  Labeling  function  is  slow.  (3  comments) 

•  Labels.  (2  comments) 

•  Labeling  takes  too  long. 

•  Labeling  targets. 

•  Direct-to  and  show  on  map. 
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How  Quickly  Were  You  Able  to  Navigate  Through 
the  Menu  Screens  on  the  MPDs**? 

Very  Quickly 
Somewhat  Quickly 
Borderline 
Somewhat  Slowly 
Very  Slowly 
0 

Pilot  Comments: 

•  No  comments  provided  by  pilots. 

How  Quickly  Were  You  Able  to  Navigate  Through 
the  Menu  Screens  on  the  SMD**? 

Very  Quickly 
Somewhat  Quickly 
Borderline 
Somewhat  Slowly 
Very  Slowly 

0%  20%  40%  60%  80%  100% 

Pilot  Comments: 

•  I  would  like  to  have  more  menu  selections  displayed  on  either  the  TIMI  or  SMD 
and  not  both.  There  is  no  logical  reason  why  a  selection  is  on  either  the  TIMI  or 
SMD. 

^Significant  at  a  .01 
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How  Quickly  Were  You  Able  to  Navigate  Through 
the  Menu  Screens  on  the  TSD**? 


Very  Quickly 
Somewhat  Quickly 
Borderline 
Somewhat  Slowly 
Very  Slowly 

0%  20%  40%  60%  80%  100% 


Pilot  Comments: 


Problems  with  labeling  of  targets: 

•  Target  labeling. 

•  Labeling  slow. 

•  The  labeling  function  and  TSD  windows  function  are  very  slow.  As  well  as 
plotting  a  direct  waypoint. 

•  Labeling  and  windows. 

•  TGT  labeling. 

•  Only  in  labeling  menus. 

•  The  menu  structure  for  TSD  labeling  is  very  poor. 

•  Threat  ID  classification  with  right  slew  hook-  too  much  time.  Should  be  choices  on 
SMD  and  TSD. 


^Significant  at  a  .01 
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How  Often  Did  You  Have  Trouble  Remembering 
Where  You  Were  in  the  MPD  Menu  System**? 


Pilot  Comments: 


•  No  comments  provided  by  pilots. 


How  Often  Did  You  Have  Trouble  Remembering 
Where  You  Were  in  the  SMD  Menu  System**? 


Pilot  Comments: 

•  No  comments  provided  by  pilots. 


"Significant  at  a  .01 


34 


How  Often  Did  You  Have  Trouble  Remembering 
Where  You  Were  in  the  TSD  Menu  System**? 


Never  Had  Trouble 
Seldom  Had  T rouble 
Often  Had  Trouble 
Constantly  Had  T rouble 

0%  20%  40%  60%  80%  100% 

Pilot  Comments: 

•  TSD  windows  can  get  confusing  sometimes.  I  like  the  purpose  of  windows, 
however  I  think  it  could  be  easier  to  use. 

**Significant  at  a  .01 
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Were  Any  Switches  on  the  Collective  Grip 
Time  Consuming  to  Use  ? 


Pilot  Comments: 


Problems  with  slew  hook  and  FOV  switches: 


•  Slew  hook  functions. 

•  Both  slew  hooks  and  FOV  zoom  select 
switch. 

•  Slew  Hook/  FOV  switches. 

•  The  FOV  selection  switch  on  the  collective 
is  not  in  logical  order.  The  optics  are 
displayed  as  follows:  MWN.  The  order 
they  should  follow  is  WMN. 

•  Field  of  view  needs  logical  order. 

•  Front/ Back:  the  FOV  selection  switch  does 
not  select  between  narrow,  medium  and 
wide  in  a  logical  order. 

Other  comments: 


•  Map  Scale-  include  18  and  32  k  scales. 

•  Front/ Back:  the  map  scale  switch  should 
also  have  option  for  18K  map  scale. 

•  The  "no  target"  and  "details"  are  difficult 
to  find  without  visual  verification.  Both 
seats. 

•  Radio  selection  -  Would  like  five  position 
switch. 


^Significant  at  a  .05 
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Were  Any  Switches  on  the  Sidearm  Controller 
Time  Consuming  to  Use*  ? 


0%  20%  40%  60%  80%  100% 


Pilot  Comments: 

•  Right  arm  controller  very  difficult  to  use. 

•  Arm  and  cyclic  inputs  difficult. 

•  Need  detachable  controller  (Nintendo®). 

•  AFCS  difficult  to  reach/ slant  control  force. 

•  Finding  the  "take  flight  control"  is 
difficult. 

•  The  SAC  needs  to  be  more  user  friendly. 
That  is,  easier  to  hold  on  to  and  use  the 
switch  at  the  same  time. 


Was  There  Any  Symbology  on  the  HMD  That 
Was  Difficult  to  Quickly  and  Easily 
Understand*? 


Was  There  Any  Symbology  on  the  SMD  That 
Was  Difficult  to  Quickly  and  Easily 
Understand**? 


f  \  11 

YES 

F  y 

77% 

NO 

23% 

W  y  : . “7  " 1  1  y1 

L“" — 7^ 

- 7-1 

0%  20%  40%  60% 

i  80% 

1  100% 

Pilot  Comments: 

Problems  with  heading  tape: 

•  Heading  tape  should  be  screen  stabilized  (6 
comments) 

•  Heading  tape  should  be  fixed  (not  moving).  (2 
comments) 

•  Heading  tape  should  be  HMD  stabilized.  (2 
comments) 

•  Heading  tape  needs  to  be  fixed  to  the  aircraft. 

•  I  still  do  not  like  the  moving  heading  tape.  It 
should  be  fixed. 

•  Heading  tape  should  be  stable. 

•  The  heading  tape  needs  to  be  stable.  It  can 
cause  vertigo. 

Other  comments: 


0%  20%  40%  60%  80%  100% 


Pilot  Comments: 

•  No  comments  provided  by  pilots. 


•  Need  to  add  pitch  ladder  (4  comments) 

•  Need  to  add  bank  angle.  (3  comments) 

•  No  TAS  LOS  bearing  indicator. 

•  False  horizon  nearly  impossible  to  use  for  any 
turn  rate  or  A/C  attitude  determination. 
Excessive  movement  of  symbology  confusing 
almost  imparting  the  desire  to  input  an 
incorrect  change  in  A/C  attitude. 

•  WCA  not  easily  identified  without  audio. 

•  Cannot  see  outside  when  operating  TAS. 

•  Symbology  for  NVS  flight  predictor  worse 
than  all  others  as  previously  stated. 

•  20  mm  ownership  not  displayed. 

•  Hellfire  ownership  not  displayed. 


*Significant  at  a  .05 
**Significant  at  a  .01 
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Was  There  Any  Symbology  on  the  TSD  That 
Was  Difficult  to  Quickly  and  Easily 
Understand*? 


0%  20%  40%  60%  80%  100% 


Pilot  Comments: 

•  18th  K  view  targets  close  to  one  another;  difficult  to  break  out  and  overlaid  symbology  selects 
when  hooked.  That  is,  hook  difficult  to  pick  between  two  different  items  that  occupy  the 
same  space  on  the  MAP. 

•  Wires  and  roads  are  red. 

•  Digital  map  symbols  are  different  than  the  chart  symbols  that  every  aviator  is  familiar  with. 
Icon  should  also  be  scaled  proportionally  to  map  scale. 

•  ADA  searching  symbol  difficult  to  see.  Sensor  shading  on  map.  Line  of  sight  with  terrain 
shading  for  line  of  sight  inter-visibility. 

•  Visual  alert  of  ADA  (ASE  radar  or  laser  warning)  on  map.  - 1  would  like  to  have  the  symbols 
red  instead  of  black.  I  would  like  to  have  it  flashing  to  catch  my  attention  if  it  is  a  new 
system  searching. 

•  The  non-labeled  vehicle  icons  are  hard  to  tell  the  difference  between  a  wheeled  and  tracked 
vehicle  icon.  They  should  look  more  like  a  truck  and  a  tank. 

•  WCA  not  noticed  without  audio  alarm  (move  to  bottom). 

•  Target  icons  in  proximity  to  graphics  and  each  other;  hard  to  separate. 
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APPENDIX  D 


COLLECTIVE,  SIDE  ARM  CONTROLLER,  AND 
RIGHT  ARMREST  SWITCHES 


Intentionally  left  blank 


COLLECTIVE,  SIDE  ARM  CONTROLLER,  AND 
RIGHT  ARMREST  SWITCHES 


Collective  switches 

Side  arm  controller  switches 

A 

Searchlight 

Trim 

Automatic  flight  control  system 

Guarded  jettison  release 

Coupler  release 

Sensor  FOV/sensor  control 

NVPS I2/IR  and  polarity  select 

Cursor  control  for  TAS  (left  slew  hook) 

Station  deselect 

No  target  designation  switch 

Trigger  guard 

Target  select  and  sensor  locking 

Weapon  release 

Location  designation  switch 

Weapon  select 

Cursor  control  for  TSD  (right  slew  hook) 

Yaw  trim  release 

Map  selection/scale  selection 

Slew  to  own  helmet  sensor  control 
Collective  trim  release 

Radio/preset  select  switch 

Warning,  caution,  advisory  acknowledge 
Laser  activation  switch 

Hover  return 

Integrated  flight-fire  control  (IFFC) 

Right  slew  hook 
Left  slew  hook 
Laser 
Details 
Find 
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Intentionally  left  blank 


APPENDIX  E 


SUMMARY  OF  PILOT  RESPONSES  TO  BEDFORD 
WORKLOAD  RATING  SCALE 


Intentionally  left  blank 


SUMMARY  OF  PILOT  RESPONSES  TO  BEDFORD 
WORKLOAD  RATING  SCALE 


Crew 

Member 

Tasks 

Front  Seat 
Workload 
Rating 

Back  Seat 
Workload 
Rating 

Crew  Member 
Tasks 

Front  Seat 
Workload 
Rating 

Back  Seat 
Workload 
Rating 

Perform 
Before  Flight 
Checks 

1.82 

2.07 

Perform  Terrain 
Flight 

Deceleration 

3.23 

NA 

Maintain 

Airspace 

Surveillance 

2.56 

a 

Perform  Masking 
And  Unmasking 

2.96 

NA 

Perform 

Radio 

Communi¬ 

cations 

3.30 

3.60 

Perform  Evasive 
Maneuvers 

4.28 

NA 

Perform 

Hovering 

Flight 

2.13 

NA 

Perform  Actions 
on  Contact 

4.00 

3.37 

Perform 
VMC  Takeoff 

2.42 

NA 

Select  Landing 
Zone /Holding 
Area 

3.20 

3.09 

Perform 
VMC  Flight 
Maneuvers 

2.58 

NA 

Perform  VMC 
Approach 

2.73 

NA 

Perform 

Electronically 

Aided 

Navigation 

2.93 

2.72 

Perform 

Inadvertent  IMC 
Procedures 

3.77b 

3.62b 

! 

Perform 
Terrain  Flight 
Navigation 

2.96 

2.66 

Perform 

Emergency 

Procedures 

3.80b 

3.20b 

Perform 
Terrain  Flight 

3.23 

NA 

Perform  TSD 
Operations 

3.21 

3.89 

aTask  could  not  be  completed  at  night  because  there  was  no  night  vision  device  for  use  while  pilots 

conducted  a  scan  with  the  TAS. 

bBased  on  ratings  from  fewer  than  10  missions. 
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Crew 

Member 

Tasks 

Front  Seat 
Workload 
Rating 

Back  Seat 
Workload 
Rating 

Crew 

Member 

Tasks 

Front  Seat 
Workload 
Rating 

Back  Seat 
Workload 
Rating 

Perform 

Target 

Handover 

2.76 

3.00 

Perform 

Digital 

Communi¬ 

cations 

4.53 

6.10 

Perform 

Firing 

Techniques 

3.76 

3.31 

Perform  Data 
Entry 

Procedures 

4.35 

6.00 

Perform 

Firing 

Position 

Operations 

3.52 

3.10 

Perform  Data 
Management 
Operations 

3.60 

4.59 

Select  A 
Combat 
Position 

3.09 

2.95 

Engage 
Target  With 
PTWS 

3.80 

3.00 

Select 

Appropriate 

Weapon 

System 

2.42 

2.75 

Engage 
Target  With 
The  AWS 
Turreted  Gun 

4.40a 

4.16a 

Perform 

HIDSS 

Boresight 

2.20 

2.48 

Perform 

Aerial 

Observation 

2.96 

3.30 

Perform 

HIDSS 

Operations 

2.60 

3.04 

Identify 

Major 

U.S. /Allied 
Equipment 
and  Major 
Threat 
Equipment 

3.29 

3.58 

Perform 

EOTADS 

Sensor 

Operations 

3.00a 

3.39 

Operate 

Aircraft 

Survivability 

Equipment 

2.70 

2.44“ 

'’Based  on  ratings  from  fewer  than  10  missions. 
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Crew  Member  Tasks 

Front  Seat 
Workload 
Rating 

BackSeat 

Workload 

Rating 

Operate  Night 
Vision  Pilotage 
System 

2.60 

3.88a 

aBased  on  ratings  from  fewer  than  10  missions. 


Comments  regarding  lack  of  external  vision  in  the  back  seat: 

•  Task  1042:  (Airspace  Surveillance)  Back  seat  has  no  vision. 

•  Task  1042:  Not  able  to  perform  airspace  surveillance  due  to  lack  of  night  vision. 

•  Task  1042:  B/S  has  no  outside  vision 

•  Task  1042:  Can’t  see  in  the  back  when  not  flying. 

•  Task  1042  cannot  be  performed  due  to  lack  of  night  vision  in  rear  seat  during 
scans. 

•  Task  1500  (Aerial  Observation):  Can't  quickly  scan  around  aircraft  (no  vision). 
Would  like  NVG  or  equivalent. 

•  Tasks  1153  (Evasive  Maneuvers),  1162  (Actions  On  Contact),  1442  (HIDSS 
Operations)  and  1448  (EOTADS  Sensor  Operations):  During  maneuvering  CP 
unable  to  acquire  TGT  with  TAS  Auto  or  manual.  Since  CP  is  HMD  blind,  the 
CP  was  unable  to  acquire  target  with  HIDSS. 

•  Task  1837  (Operate  Night  Vision  Pilotage  System):  Not  available,  so  abandoned 
flight  to  FS. 

•  I  have  no  visual  in  the  back 

Comments  regarding  the  CIK: 

•  Tasks  1449  (Digital  Communications  and  1454  (Data  Entry  Procedures):  CIK  very 
cumbersome.  (2  comments) 

•  Digital  communications  are  totally  absorbing. 

•  Task  1100  (Radio  Communications):  CIK  very  cumbersome 

•  The  CIK  is  not  user  friendly  and  too  slow. 

•  Using  the  CIK  and  using  12  is  cumbersome.  The  whole  digital  processing  needs 
work. 

•  The  CIK  needs  to  be  more  user-friendly. 

Comments  regarding  other  cockpit  interface  problems: 

•  Task  1410  (TSD  Operations)  Controls  are  difficult  to  operate  which  causes 
workload  to  be  higher. 

•  TAS  view  last,  multi-step  must  complete  before  continuing  with  other  functions. 

•  Data  procedures  require  excessive  number  of  steps  and  are  difficult  to  work  with 
current  controls. 

•  HIDSS  symbology:  Reference  to  aircraft  attitude  and  turn  angle  difficult  to 
determine  from  looking  off  nose  or  slides.  Determining  rate  of  turn  impossible. 
Just  guessing  at  it  from  movement  of  heading  tape.  Predictor  a  waste  of  space, 
pilot  must  fly  terrain. 

•  1449  (Digital  Communications)  and  1455  (Data  Management  Operations):  Vocal 
reports  used  to  augment  diagram  because  of  poor  interface. 

•  Task  1455  (Data  Management  Operations):  Labeling  targets  cumbersome. 
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•  Tasks  1410  (TSD  Operations,  1442  (HIDSS  Operations),  1449  (Digital 
Communications),  1454  (Data  Entry  Procedures),  1455  (Data  Management 
Operations)  and  1837  (Operate  Night  Vision  Pilotage  System):  System  interface  is 
cumbersome  requiring  excessive  time. 

•  Task  1162  (Actions  On  Contact)  -  Only  focus  was  flying  the  A/C. 

•  Tasks  1422  (Firing  Techniques)  and  1426  (Firing  Position  Operations):  Wouldn't 
allow  an  engagement. 

•  When  the  MPED  would  direct  WP  the  NAI,  I  could  not  see  the  line  on  my  TSD. 

•  Had  to  wait  a  long  time  for  target  ID. 

•  TSD/TAS  labeling  due  to  ATS  not  Id'ing  targets  5K  and  under.  INT  not 
functioning  properly. 

•  Encountered  ADA  and  had  to  quickly  mask  without  crashing. 

•  Task  1442  (HIDSS  Operations):  Not  used  for  other  than  missile  constraints. 

•  Unable  to  load  and  select  presets. 
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Rate  Whether  Overall  Workload  was  Higher  for 
the  Front  Seat  or  Back  Seat**? 


FS  Workload  Was  Much  Higher 
FS  Workload  Was  Somewhat  Higher 
About  The  Same 
BS  Workload  Was  Somewhat  Higher 
BS  Workload  Was  Much  Higher 

0%  20%  40%  60%  80%  100% 


Pilot  comments  on  why  they  experienced  higher  workload  in  the  back  seat: 

Comments  regarding  operation  of  the  MEP: 

•  The  MEP  operator  usually  has  higher  workload. 

•  MEP  operator  (has  higher  workload). 

•  Running  the  MEP  is  higher  workload. 

•  As  MEP  operator,  it  requires  a  lot  of  attention. 

•  Back  seat  (had  higher  workload)  because  of  working  the  majority  of  the  MEP  in 
this  scenario. 

•  Current  configuration  places  the  MEP  operator  in  the  rear  seat. 

•  The  back  seat  was  used  as  the  MEP  operator. 

•  More  MEP  operations  (in  the  back  seat). 

•  Because  the  back  seat  has  to  be  able  to  communicate,  ID  the  target  and  maintain 
airspace  surveillance.  All  at  the  same  time. 

•  The  front  seater  just  flies,  the  back  seater  was  doing  all  the  work. 

•  The  individual  not  flying  classifies  and  communicates  and  services  while  the  front 
seat  flew. 

•  More  mission  requirement  in  back  seat 

•  Front  seat  responsible  for  obs  avoidance,  NAV,  radio  traffic.  Rear  seat  threat 
location,  ID,  spot  reports. 

•  It  does  take  a  lot  of  effort  to  fly  point  to  point,  the  back  seat  scanning  and  reporting 
ARTY,  weapons  engagement,  (all  "inside"  work). 


^^Significant  at  a  .01 
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Comments  regarding  the  TAS: 

•  Back  seat  needs  NVG  or  some  system  to  see  outside  the  A/C  when  the  TAS  is 
used. 

•  TAS  manual  slew  for  scanning  or  viewing  difficult. 

•  P  (not  on  controls)  needs  vision  to  improve  TAS  operations. 

•  Rear  seat  should  be  able  to  see  outside  the  aircraft  even  when  the  TAS  is  operated. 

•  Slew  hook  for  TAS  manual  ops  does  not  allow  for  any  real  TGT,  ID  or  develop¬ 
ment.  No  azimuth,  no  target,  and  no  range. 

•  Front  seat  primarily  responsible  for  flying,  obstacle  avoidance,  NAV.  Rear  seat  all 
systems  with  TAS  and  NPVs. 

Comments  regarding  labeling  function: 

•  The  labeling  function  is  too  slow. 

•  Need  new  labeling  system  for  TGTS. 

•  Target  labeling  in  TSD. 

•  Labels/Windows. 

•  TAS/Labeling/ Windows  requires  high  workload  with  eyes  in  almost  all  times. 

Comments  regarding  radio  communications: 

•  Manual  entry  of  frequencies  and  grid  locations  to  navigate  with. 

•  Selecting  radio  requires  visual  verification  (hands  on).  Radio  frequency  display 
position  is  far  from  center. 

•  Location  of  RFD  makes  it  difficult  to  see  and  read.  Need  ability  to  see  which  of 
our  five  radios  people  are  transmitting  on  to  aid  in  situational  awareness. 

Comments  regarding  the  CIK: 

•  B/S  ops  on  CIK.  The  keyboard  is  cumbersome.  I  would  like  it  in  a  normal  typing 
configuration  with  the  numbers  in  a  numerical  keypad  configuration. 

•  The  CIK  is  too  slow. 

Other  comments: 

•  There  needs  to  be  another  trigger  for  WPNS  engagement. 

•  The  heading  tape  should  remain  stationary.  It  can  be  confusing  if  you  look  down 
then  back  up  and  everything  is  moving.  HSI  indicators  are  hard  to  see.  Both  crew 
stations  are  looking  inside  while  no  one  is  focusing  outside. 

•  Spot  reports,  difficult  to  separate  targets  destroyed  from  those  not  destroyed. 
Sometimes  difficult  to  separate  line  and  rotated  targets. 

Pilot  comments  on  why  they  experienced  higher  workload  in  the  Front  Seat: 

•  FS  only  because  of  difficulty  of  cross  checking  dash  displays  with  viewing  out  the 
window  (HMD)  and  flying  very  low  and  fast. 

•  ASE  display  was  difficult  to  view  and  avoid  the  terrain  from  F.S. 
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Crew  workload  was  assessed  during  the  RAH-66  Comanche  Force  Development  Experiment  (FDE)  1.  The  purpose  was  to 
determine  if  (a)  the  pilots  experienced  tolerable  and  comparable  workload  levels  when  flying  the  aircraft  versus  operating  the 
mission  equipment  package  (MEP),  and  (b)  workload  levels  contribute  to  a  need  to  "battle  roster"  Comanche  pilots.  Workload 
data  were  collected  via  the  Bedford  Workload  Rating  Scale  and  a  cockpit  controls  and  displays  usability  questionnaire. 

Results  of  the  assessment  indicate  that  (a)  workload  was  tolerable  for  the  pilots,  (b)  workload  was  moderately  higher  for  the 
pilots  when  they  operated  the  MEP,  and  (c)  there  is  no  compelling  need  to  prescribe  battle  rostering  because  pilots  experienced 
only  moderate  differences  in  workload  when  they  flew  the  aircraft  versus  when  they  operated  the  MEP. 
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